Image processing apparatus, image processing method, and image sensing apparatus

ABSTRACT

An image processing apparatus includes a spectrum estimation unit for estimating a spectrum for each pixel from multiband image data comprising image data in multiple different wavelength ranges; a memory for storing spectral data suitable for each of multiple different shooting modes; and a conversion unit for determining a shooting mode of the multiband image data and converting the spectrum into band data with different spectral characteristics using the spectral data stored in the memory, the spectral data suitable for the determined shooting mode.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and an image sensing apparatus for processingmultiband image data, and more particularly, to an image processingapparatus, an image processing method, and an image sensing apparatusfor converting multiband image data into image data with differentspectral characteristics.

2. Description of the Related Art

Multiband image sensing apparatuses (for example, multiband cameras,etc.) have recently begun to be commercialized, from which spectralimages of objects can be obtained by shooting the objects with the useof multiple filters that transmit different wavelength ranges. In orderto see the multiband images obtained from the multiband image sensingapparatus with the use of an ordinary output device, the multiband imagedata has to be converted to RGB 3-band data, for instance.

As the conversion method, there are techniques for conversion into3-channel, for example, RGB image data by selecting necessary channelsfrom among multi-channel image data in accordance with the purpose ofshooting (for example, see Japanese Patent Laid-Open No. 2001-78226).The use of this technique allows images of 3-channel image data suitablefor the purpose of shooting to be output.

On the other hand, for image sensing apparatuses, such as digital stillcameras, with conventional primary color RGB filters or the like, RGBcolor filters are used which have spectral transmittance characteristicssuch that optimum images can be obtained in a color space of sRGB.Recently, output devices such as displays and printers, for color spaceslarger than sRGB, such as AdobeRGB, have also been developed. Therefore,for images output by the image sensing apparatus that is an inputdevice, there is also a need for outputting images for color spaceslarger than sRGB, such as AdobeRGB.

However, the color filters optimized for color reproduction in sRGB havedifficulty obtaining image data for large color spaces such as AdobeRGB.

However, as described in Japanese Patent Laid-Open No. 2001-78226, in acase in which conversion into 3-channel image data is carried out byselecting necessary channels from among multi-channel image dataobtained with the use of multiple filters that have different spectraltransmittance characteristics, the channel data that can be selected islimited. Furthermore, the white balance of the multi-channel image datais not adjusted in the technique described in Japanese Patent Laid-OpenNo. 2001-78226. Therefore, the white balance has to be adjusted afterconversion into 3-channel data, depending on 3-channel spectra used forthe creation of the 3-channel images.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and has as its object to increase the degree of freedom ofconverting multiband image data into different image data in the numberof bands. It is another object of the present invention to eliminate thetrouble of having to change the white balance depending on theconversion method.

According to the present invention, the foregoing object is attained byproviding an image processing apparatus comprising: an estimation unitconfigured to estimate a spectrum for each pixel from multiband imagedata comprising image data in multiple different wavelength ranges; astorage unit configured to store spectral data suitable for each ofmultiple different shooting modes; and a conversion unit configured todetermine a shooting mode of the multiband image data and convert thespectrum into band data with different spectral characteristics usingthe spectral data stored in the storage unit, the spectral data suitablefor the determined shooting mode.

Further, according to the present invention, the foregoing object isalso attained by providing an image processing method comprising: anestimation step of estimating a spectrum for each pixel from multibandimage data comprising image data in multiple different wavelengthranges; a determination step of determining a shooting mode of themultiband image data; and a conversion step of selecting spectral datasuitable for the shooting mode determined in the determination step froma storage unit configured to store spectral data suitable for each ofmultiple different shooting modes, and converting the spectrum into banddata with different spectral characteristics using the selected spectraldata.

Furthermore, the foregoing object is also attained by providing an imageprocessing apparatus comprising: an estimation unit configured toestimate a spectrum for each pixel from multiband image data comprisingimage data in multiple different wavelength ranges; a storage unitconfigured to store spectral data suitable for each of multipledifferent color spaces; a setting unit configured to set one of themultiple color spaces; and a conversion unit configured to convert thespectrum into band data with different spectral characteristics usingthe spectral data stored in the storage unit, the spectral data suitablefor the color space set by the setting unit.

Further, the foregoing object is also attained by providing an imageprocessing method comprising: an estimation step of estimating aspectrum for each pixel from multiband image data comprising image datain multiple different wavelength ranges; a setting step of setting oneof multiple color spaces; a conversion step of selecting spectral datasuitable for the color space set in the setting step from a storage unitconfigured to store spectral data suitable for each of the multiplecolor spaces, and converting the spectrum into band data with differentspectral characteristics using the selected spectral data.

Furthermore, the foregoing object is also attained by providing an imagesensing apparatus comprising: multiple spectral filters with spectraltransmittance characteristics different from each other; an imagesensing unit configured to photoelectrically convert light transmittedthrough each of the spectral filters to output multiband image datacomprising image data in multiple different wavelength ranges; and oneof the foregoing image processing apparatuses.

Further, features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of multiband spectraltransmittance characteristics according to first and second embodimentsof the present invention;

FIG. 2 is a diagram illustrating a multiband turret according to thefirst and second embodiments of the present invention;

FIG. 3 is a diagram schematically illustrating the configuration of animage sensing apparatus according to the first and second embodiments ofthe present invention;

FIG. 4 is a diagram illustrating an example of a spectral filterarrangement disposed on an image sensor according to the first andsecond embodiments of the present invention;

FIG. 5 is a diagram illustrating the functional configuration of theimage sensing apparatus according to the first and second embodiments ofthe present invention;

FIG. 6 is a diagram showing an example of a spectrum of an image formedby synthesized light of reflected light from an object and light from alight source according to the first and second embodiments of thepresent invention;

FIG. 7 is a diagram showing an example of image data obtained from lightwith the spectrum shown in FIG. 6;

FIG. 8 is a flowchart showing a method for calculating WB correctioncoefficients in multiband image processing according to the first andsecond embodiments of the present invention;

FIG. 9 is a diagram showing an example of a spectrum of a referenceimage according to the first and second embodiments of the presentinvention;

FIG. 10 is a diagram showing an example of pixel data obtained fromlight with the spectrum shown in FIG. 9;

FIG. 11 is a diagram showing multiband pixel data after WB correctionaccording to the first and second embodiments of the present invention;

FIG. 12 is a diagram for explaining a method of estimating a spectrum ofan object from multiband pixel data according to the first and secondembodiments of the present invention;

FIG. 13 is a diagram showing a spectrum of a Hα line;

FIG. 14 is a diagram showing spectral data to be used in a first modeaccording to the first embodiment of the present invention;

FIG. 15 is a diagram showing spectral data to be used in a second modeaccording to the first embodiment of the present invention;

FIG. 16 is a flowchart showing the flow of multiband image processingaccording to the first embodiment of the present invention;

FIG. 17 is a diagram showing spectral data for sRGB according to thesecond embodiment of the present invention;

FIG. 18 is a diagram showing spectral data for AdobeRGB according to thesecond embodiment of the present invention; and

FIG. 19 is a flowchart showing the flow of multiband image processingaccording to the second embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

In the first embodiment, a method of converting multiband image datainto three-band image data in accordance with the purpose of shootingwill be described.

FIG. 1 shows multiband spectral transmittance characteristics used inthe first embodiment. In the present embodiment, an image sensingapparatus will be described using 9-band spectral filters which havespectral transmittance characteristics mainly in each of wavelengthranges (a) to (i) as shown in FIG. 1. However, it is to be noted thatthe number of bands used or the type of spectral transmittancecharacteristics is not limited to the embodiment. It is to be noted thatthe correspondences of the wavelength ranges (a) to (i) to spectralfilters and data will be represented by the symbols (a) to (i) in thefollowing description.

The spectral filters (a) to (i) which have each of the spectraltransmittance characteristics shown in FIG. 1 are, for example,regularly arranged on a turret, as filters (a) to (i) shown in FIG. 2.How to use this turret will be described below with reference to FIG. 3.

In FIG. 3, reference numerals 301, 302, 303, 304, and 305 respectivelydenote an object, a light source for illuminating the object, themultiband turret shown in FIG. 2 (hereinafter referred to as a turret),a lens, and an image sensor such as a CCD sensor or a CMOS sensor. Thelens 304 is disposed so that an image of the object transmitted throughone of the spectral filters (a) to (i) on the turret 303 is formed onthe image sensor 305. The image of the object entering the image sensor305 is converted into electrical signals by photoelectric conversion,and output. It is to be noted that the turret 303 disposed in front ofthe lens 304 is rotated every time shooting is carried out, andcontrolled so that an image of the object transmitted through adifferent spectral filter enters the lens 304. When shooting is repeatednine times while changing the spectral filters, image data (a) to (i) inthe wavelength ranges (a) to (i) is output.

Moreover, multispectral image data may be output, for example, byarranging the spectral filters (a) to (i) on a single-plate image sensor(for example, a CCD sensor or a CMOS sensor) as shown in FIG. 4, inplace of the spectral filters (a) to (i) arranged on the turret.

Next, the flow of processing carried out in the multiband imageprocessing apparatus in the first embodiment will be described, wherethe turret 303 is used.

FIG. 5 is a block diagram illustrating the configuration of an imageprocessing apparatus according to the first embodiment.

In FIG. 5, reference numeral 305 denotes the image sensor described withreference to FIG. 3. It is to be noted that the object side of the imageprocessing apparatus with respect to the image sensor 305 has the sameconfiguration as in FIG. 3, where the lens 304 and the turret 303 aredisposed.

Reference numeral 501 denotes an A/D converter for converting analogelectrical signals obtained through conversion by the image sensor 305,into digital signals.

Reference numeral 502 denotes a multiband image synthesizing unit forsynthesizing A/D converted image data (a) to (i) for each band of thewavelength ranges (a) to (i). In the synthesis processing carried outhere, the nine shot images are synthesized so that image data for eachof the wavelength ranges (a) to (i) is collected for each pixel and eachpixel has data for each of the wavelength ranges (a) to (i). Forexample, in a case in which an image of the object 301 entering onepixel of the image sensor 305 has a spectrum as shown in FIG. 6, thepixel has such data (pixel data) as pixel data (a) to (i) shown in FIG.7 after synthesis by the multiband image synthesizing unit 502.

The pixel data shown here in FIG. 7 contains data attributed toreflected light from the object 301 and light entering directly from thelight source 302. Therefore, it is a white balance processing unit (WBprocessing unit) 503 that eliminates the effect of the light source 302from this image data. Carrying out WB correction eliminates the effectof the light source 302 to obtain multiband image data indicative ofonly the reflectivity of the object. The method for the white balancecorrection (WB correction) will be described later.

Reference numeral 504 denotes a spectrum estimation unit for estimatinga spectrum for the object from the multiband image data subjected to theWB correction. The method for estimating a spectrum for the object fromthe image data for 9 bands will be described later.

Reference numeral 505 denotes a conversion unit for integrating theestimated spectrum of the object with predetermined spectralcoefficients for RGB to carry out conversion into RGB 3-band data. Inthe first embodiment, a plurality of spectral coefficients for RGB eachcorresponding to different shooting modes are set in advance, andspectral coefficients corresponding to a shooting mode set by the userin advance will be used. The shooting mode and the correspondingspectral data will be described later.

Reference numeral 506 denotes an image processing unit for performing,for the RGB 3-band data, the gamma processing, and adjusting color hue,saturation, brightness, and the like generally used in conventional RGBband image sensing apparatuses. Only after these types of processing areall carried out will RGB 3-band images be created from the multibandimages.

Next, a method for calculating WB correction coefficients to be used forWB processing in the WB processing unit 503 will be described in detailwith reference to a flowchart in FIG. 8.

First, in order to obtain data on the light source as the basis for WBcorrection coefficients for white balance, for example, a grey uniformluminance surface with a reflectivity of 18% is shot as a referenceimage under the light source 302 illuminating the object (step S11).Here, light from the light source 302 is assumed to have such spectralcharacteristics as in FIG. 9. The reference image data obtained byshooting the reference image is subjected to A/D conversion by the A/Dconverter 501 (step S12). The A/D converted image data for one band istemporarily recorded in a memory (not shown) in the image sensingapparatus in order to be later synthesized into multiband image data(step S13).

In step S14, it is determined whether or not the processing of the stepsS11 to S13 is carried out for all of the spectral filters (a) to (i)arranged on the turret 303. Until processing is completed with the useof all of the spectral filters (a) to (i), the processing of the stepS11 to S13 is repeatedly carried out while changing the spectral filters(step S15).

When the processing is completed with the use of all of the spectralfilters (a) to (i) arranged on the turret 303 (YES in step S14), theimages for all of the bands, which have been recorded in the memory inthe image sensing apparatus, are synthesized into one image by themultiband image synthesizing unit 502 (step S16). This synthesized imagebecomes image data (a) to (i) having pixel data (a) to (i) respectivelycorresponding to the wavelength ranges (a) to (i) for each pixel, asshown in FIG. 10.

Then, the WB correction coefficients with which the image data (a) to(i) is integrated are determined such that the average value of thepieces of image data (d), (e), and (f) among the 9 pieces of image data(a) to (i) takes the value of 1 (step S17).

Specifically, the WB correction coefficients are calculated inaccordance with the following formulas:

WB correction coefficient for wavelength range (a)=Va/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (b)=Vb/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (c)=Vc/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (d)=Vd/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (e)=Ve/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (f)=Vf/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (g)=Vg/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (h)=Vh/((Vd+Ve+Vf)/3)

WB correction coefficient for wavelength range (i)=Vi/((Vd+Ve+Vf)/3)

In the above-mentioned formulas, Va to Vi represent the average valuesor integration values of pixel data (a) to (i) for every pixelrespectively corresponding to the wavelength ranges (a) to (i).

In the first embodiment, the WB correction coefficients are calculatedusing the image data (d), (e), and (f) as a reference. However, thereference image data may be any one piece of the image data (a) to (i),or a combination of multiple pieces thereof as in the first embodiment.

The calculated WB correction coefficients are stored in the memory ofthe image sensing apparatus (step S18) to complete calculation of the WBcorrection coefficients.

The WB processing unit 503 subjects each piece of the pixel data (a) to(i) for each pixel, attributed to reflected light from the object 301shown in FIG. 7 and light entering directly from the light source 302 asdescribed above, to WB correction by integration with the WB correctioncoefficients calculated in advance. As a result, the effect of the lightsource 302 can be eliminated to obtain pixel data (a) to (i) based ononly the reflected light from the object 301, as shown in FIG. 11.

Next, the processing carried out in the spectrum estimation unit 504will be described in detail with reference to FIG. 12, which showsresultant outputs after the WB correction.

In FIG. 12, reference numeral 1201 denotes a spectrum for the object 301estimated from the pixel data (a) to (i). This estimated spectral data1201 can be estimated by connecting dots 1202 to 1210 at centralwavelengths for each piece of the pixel data (a) to (i), like a curve ofthe second order. It is to be noted that the spectrum estimation is notlimited to the method described above, and thus, for example, Wienerspectrum estimation method which is commonly known or the like may beused.

Next, a method will be described in detail, wherein a method forconverting the estimated spectrum of the object into RGB 3-band data ischanged depending on a shooting mode set in advance.

The image sensing apparatus according to the present embodiment has thefunction of setting different shooting modes (not shown), and in thefirst embodiment, is assumed to have a first mode of faithfullyoutputting the colors of the object and a second mode suitable forastronomical shooting. It is to be noted that the number and types ofshooting modes are not limited to this assumption. The first mode is amode for faithfully reproducing colors recognized by the human eyes. Onthe other hand, the second mode is a mode suitable for astronomicalshooting. In astronomical shooting, objects to be shoot often include aHa line, and it is preferred that the portions including the Ha line behighlighted by redness as astronomical shot images. This Hα line isknown to have an emission line in a wavelength range of 656.3 nm, asshown in FIG. 13.

In a case in which the first shooting mode is selected from the twotypes of shooting modes, RGB 3-band data is obtained by integrating theestimated spectrum for the object with RGB spectral data equivalent tocolor matching function considered close to spectra of the human eye, asshown in FIG. 14.

Alternatively, in a case in which the second shooting mode is selected,RGB 3-band data is obtained by integrating the estimated spectrum forthe object with RGB spectral data that provides relatively strong Rsensitivity around the Hα line (656.3 nm) with respect to G and Bsensitivities, as shown in FIG. 15.

It is to be noted that although conversion into RGB 3-band data isdescribed in the first embodiment, conversion into four or more banddata may be carried out. In such a case, the conversion can be achievedby integrating the estimated spectrum for the object with pieces ofspectral data equal in number to the required number of bands.

Next, a series of steps of the processing described above will bedescribed with reference to a flowchart shown in FIG. 16. It is to benoted that the WB correction coefficients described with reference tothe flowchart of FIG. 8 is assumed to be obtained prior to theprocessing.

First, the user selects any of the shooting modes provided in the imagesensing apparatus (step S21). Next, shooting is carried out (step S22),and image data output from the image sensor 305 is subjected to A/Dconversion by the A/D converter 501 (step S23). The A/D converted imagedata for one band is temporarily recorded in a memory (not shown) in theimage sensing apparatus in order to be later synthesized into multibandimage data (step S24).

In step S25, it is determined whether or not the processing of the stepsS22 to S24 is carried out for all of the spectral filters (a) to (i)arranged on the turret 303. Until processing is completed with the useof all of the spectral filters (a) to (i), the processing of the stepS22 to S24 is repeatedly carried out while changing the spectral filters(step S26).

When the processing is completed with the use of all of the spectralfilters (a) to (i) arranged on the turret 303 (YES in step S25), theprocess proceeds to step S27. Then, the images for all of the bands,which have been recorded in the memory in the image sensing apparatus,are synthesized into one image by the multiband image synthesizing unit502 as described above. This synthesis will provide each pixel withpixel data (a) to (i).

Next, each piece of the pixel data (a) to (i) obtained by the synthesisis subjected to WB correction by integration with a WB correctioncoefficient, calculated in advance, for the corresponding wavelengthrange (step S28). From the pixel data (a) to (i) subjected to the WBcorrection, the spectrum for the object is estimated for each pixel withthe use of the approach described above (step S29). Then, the estimatedspectrum for each pixel is integrated with RGB spectral datacorresponding to the shooting mode (the first or second mode) set instep S21 to obtain RGB 3-band data (step S30), where the spectral datashown in FIG. 14 is used in a case in which the first mode is setwhereas the spectral data shown in FIG. 15 is used in a case in whichthe second mode is set.

Finally, the RGB 3-band data is subjected to image processing for thegamma correction, and adjustment of color hue, saturation, andbrightness generally used in conventional RGB band image sensingapparatuses to complete the series of processing steps (step S31).

As described above, according to the first embodiment, the preparationof suitable spectral data for each different shooting mode in advanceallows conversion into RGB band data in accordance with the user'spurpose of shooting.

Second Embodiment

Next, a second embodiment of the present invention will be described. Inthe second embodiment, a method will be described for convertingmultiband image data into 3-band data in accordance with the color spaceinstead of the shooting mode.

It is to be noted that the multiband image synthesis method, WBcorrection coefficient calculation method, spectrum estimation method,and the like in the second embodiment are similar to those in the firstembodiment, and descriptions thereof will be thus omitted. Here, theembodiment will describe only how an estimated spectrum for an object isconverted into RGB data in accordance with the color space.

In the second embodiment, an explanation will be given on the assumptionthat two types of color spaces, sRGB and AdobeRGB, can be set as thecolor space. It is to be noted, however, that the types and settablenumber of color spaces is not limited to this assumption.

FIG. 17 shows spectral data to be used when the sRGB is set, with whichan estimated spectrum of an object is integrated, whereas FIG. 18 showsspectral data to be used when the AdobeRGB is set. The spectral datashown in FIG. 17 is spectral data optimized for color reproduction insRGB. On the other hand, in contrast to FIG. 17, the spectral data inFIG. 18 has larger separation widths set for spectra of RGB so thatcolor reproduction in a larger color space is more easily carried outthan in sRGB.

In the second embodiment, the conversion into RGB 3-band data isdescribed. It is to be noted, however, that conversion into four or moreband data may be carried out. In such a case, the conversion can beachieved by integrating the estimated spectrum of the object with piecesof spectral data equal in number to the required number of bands.

Next, a series of steps of the processing in the second embodiment willbe described with reference to a flowchart shown in FIG. 19. It is to benoted that the WB correction coefficients described in the firstembodiment with reference to the flowchart of FIG. 8 are assumed to beobtained in advance prior to the processing. In addition, the sameprocessing steps as those in FIG. 16 are denoted by the same referencenumerals, and descriptions thereof will be omitted appropriately.

First, the user selects any of the color spaces provided in the imagesensing apparatus (step S41). Next, shooting is carried out with the useof each of the spectral filters (a) to (i), and processing steps up tospectrum estimation are carried out for each pixel (steps S22 to S29).

Then, the estimated spectrum for each pixel is integrated with RGBspectral data corresponding to the color space (sRGB or AdobeRGB) set instep S41 to obtain RGB 3-band data (step S42), where the spectral datashown in FIG. 17 is used in a case in which sRGB is selected whereas thespectral data shown in FIG. 18 is used in a case in which AdobeRGB isselected.

Finally, the RGB 3-band data is subjected to image processing for thegamma correction, and adjustment of color hue, saturation, andbrightness generally used in conventional RGB band image sensingapparatuses to complete the series of processing steps (step S31).

As described above, according to the second embodiment, the preparationof suitable spectral data for each different color space in advanceallows conversion into RGB band data in a desired color space.

It is to be noted that a case of shooting nine images with the use ofthe turret 303 and synthesizing the images to obtain the image data (a)to (i) for each pixel has been described in the first and secondembodiments. In a case in which the filters (a) to (i) are arranged onthe image sensor 305 as shown in FIG. 4, it is contemplated that imagesobtained by shooting once will be processed in units of nine pixels. Inaddition, control may be exercised so as to provide each pixel withpixel data in the wavelength ranges (a) to (i) by carrying out shootingnine times using so-called pixel shifting technique in which shootingwhile displacing the position of an image of the object on the imagesensor 305. Alternatively, interpolation processing may be carried outso as to provide each pixel with pixel data in the wavelength ranges (a)to (i)

It is to be noted that although a case of reducing the nine bands intothree bands has been described in the first and second embodiments, thepresent invention is not limited to a case of reducing the number ofbands. Thus, for example, it is also possible to increase the number ofbands by integrating an estimated spectrum with spectral data for thelarger number of bands.

Other Embodiments

Note that the present invention can be applied to an apparatuscomprising a single device (a digital still camera, a digital videocamera, and so forth) or to system constituted by a plurality of devices(a host computer, an interface device, a camera head, a scanner, and soforth).

Furthermore, the invention can be implemented by supplying a softwareprogram, which implements the functions of the foregoing embodiments,directly or indirectly to a system or apparatus, reading the suppliedprogram code with a computer of the system or apparatus, and thenexecuting the program code. In this case, so long as the system orapparatus has the functions of the program, the mode of implementationneed not rely upon a program.

Accordingly, since the functions of the present invention areimplemented by computer, the program code installed in the computer alsoimplements the present invention. In other words, the claims of thepresent invention also cover a computer program for the purpose ofimplementing the functions of the present invention.

In this case, so long as the system or apparatus has the functions ofthe program, the program may be executed in any form, such as an objectcode, a program executed by an interpreter, or script data supplied toan operating system.

Examples of storage media that can be used for supplying the program area floppy disk, a hard disk, an optical disk, a magneto-optical disk, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, a non-volatile type memorycard, a ROM, and a DVD (DVD-ROM and a DVD-R).

As for the method of supplying the program, a client computer can beconnected to a website on the Internet using a browser of the clientcomputer, and the computer program of the present invention or anautomatically-installable compressed file of the program can bedownloaded to a recording medium such as a hard disk. Further, theprogram of the present invention can be supplied by dividing the programcode constituting the program into a plurality of files and downloadingthe files from different websites. In other words, a WWW (World WideWeb) server that downloads, to multiple users, the program files thatimplement the functions of the present invention by computer is alsocovered by the claims of the present invention.

It is also possible to encrypt and store the program of the presentinvention on a storage medium such as a CD-ROM, distribute the storagemedium to users, allow users who meet certain requirements to downloaddecryption key information from a website via the Internet, and allowthese users to decrypt the encrypted program by using the keyinformation, whereby the program is installed in the user computer.

Besides the cases where the aforementioned functions according to theembodiments are implemented by executing the read program by computer,an operating system or the like running on the computer may perform allor a part of the actual processing so that the functions of theforegoing embodiments can be implemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or a part of the actual processing so thatthe functions of the foregoing embodiments can be implemented by thisprocessing.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions. This application claims the benefit of Japanese PatentApplication No. 2007-274336, filed on Oct. 22, 2007, which is herebyincorporated by reference herein in its entirety.

1. An image processing apparatus comprising: an estimation unitconfigured to estimate a spectrum for each pixel from multiband imagedata comprising image data in multiple different wavelength ranges; astorage unit configured to store spectral data suitable for each ofmultiple different shooting modes; and a conversion unit configured todetermine a shooting mode of the multiband image data and convert thespectrum into band data with different spectral characteristics usingthe spectral data stored in the storage unit, the spectral data suitablefor the determined shooting mode.
 2. The image processing apparatusaccording to claim 1, further comprising a white balance processing unitconfigured to apply white balance correction to the multiband image dataprior to estimation of the spectrum by the estimation unit.
 3. The imageprocessing apparatus according to claim 1, wherein the conversion unitcarries out conversion by integrating the spectrum with the spectraldata.
 4. An image sensing apparatus comprising: multiple spectralfilters with spectral transmittance characteristics different from eachother; an image sensing unit configured to photoelectrically convertlight transmitted through each of the spectral filters to outputmultiband image data comprising image data in multiple differentwavelength ranges; and the image processing apparatus according toclaim
 1. 5. An image processing apparatus comprising: an estimation unitconfigured to estimate a spectrum for each pixel from multiband imagedata comprising image data in multiple different wavelength ranges; astorage unit configured to store spectral data suitable for each ofmultiple different color spaces; a setting unit configured to set one ofthe multiple color spaces; and a conversion unit configured to convertthe spectrum into band data with different spectral characteristicsusing the spectral data stored in the storage unit, the spectral datasuitable for the color space set by the setting unit.
 6. The imageprocessing apparatus according to claim 5, further comprising a whitebalance processing unit configured to apply white balance correction tothe multiband image data prior to estimation of the spectrum by theestimation unit.
 7. The image processing apparatus according to claim 5,wherein the conversion unit carries out conversion by integrating thespectrum with the spectral data.
 8. An image sensing apparatuscomprising: multiple spectral filters with spectral transmittancecharacteristics different from each other; an image sensing unitconfigured to photoelectrically convert light transmitted through eachof the spectral filters to output multiband image data comprising imagedata in multiple different wavelength ranges; and the image processingapparatus according to claim
 5. 9. An image processing methodcomprising: an estimation step of estimating a spectrum for each pixelfrom multiband image data comprising image data in multiple differentwavelength ranges; a determination step of determining a shooting modeof the multiband image data; and a conversion step of selecting spectraldata suitable for the shooting mode determined in the determination stepfrom a storage unit configured to store spectral data suitable for eachof multiple different shooting modes, and converting the spectrum intoband data with different spectral characteristics using the selectedspectral data.
 10. An image processing method comprising: an estimationstep of estimating a spectrum for each pixel from multiband image datacomprising image data in multiple different wavelength ranges; a settingstep of setting one of multiple color spaces; a conversion step ofselecting spectral data suitable for the color space set in the settingstep from a storage unit configured to store spectral data suitable foreach of the multiple color spaces, and converting the spectrum into banddata with different spectral characteristics using the selected spectraldata.
 11. A computer-readable storage medium storing a program forcausing a computer to execute each step of the image processing methodaccording to claim
 9. 12. A computer-readable storage medium storing aprogram for causing a computer to execute each step of the imageprocessing method according to claim 10.